Overlaying 3d augmented reality content on real-world objects using image segmentation

ABSTRACT

Various embodiments are generally directed to techniques of overlaying a virtual object on a physical object in augmented reality (AR). A computing device may receive one or more images of the physical object, perform analysis on the images (such as image segmentation) to generate a digital outline, and determine a position and a scale of the physical object based at least in part on the digital outline. The computing device may configure (e.g., rotate, scale) a 3D model of the physical object to match the determined position and scale of the physical object. The computing device may place or overlay a 3D virtual object on the physical object in AR based on a predefined location relation between the 3D virtual object and the 3D model of the physical object, and further, generate a composite view of the placement or overlay.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.17/114,983, filed on Dec. 8, 2020, which is a continuation of U.S.patent application Ser. No. 16/252,869, filed on Jan. 21, 2019 (issuedas U.S. Pat. No. 10,867,447 on Dec. 15, 2020). The contents of both theaforementioned patent applications are incorporated herein by referencein their entireties.

BACKGROUND

Augmented reality (AR) is an interactive experience of a real-worldenvironment whereby the objects that reside in the real-world areaugmented by overlaying computer-generated perceptual information. Theoverlaid information may be additive to the natural environment.

There are several AR techniques to render virtual images over real-worldobjects. Simultaneous Localization and Mapping (SLAM) is an AR techniquethat localizes sensors with respect to their surroundings, while at thesame time maps the structure of the environment. Recognition-based (ormarker-based) AR uses a camera to identify visual markers or objects toshowcase an overlay only when the marker is sensed by the device.Location-based AR relies on GPS, a digital compass, a velocity meter, oran accelerometer to provide data about location, and the ARvisualizations are activated based on these inputs.

However, neither the above-described AR techniques nor other known ARtechniques allow three-dimensional (3D) virtual objects to be overlaidor lined-up with real-world objects in a precise manner. Accordingly,there is a need for overlaying 3D virtual objects on real-world objectsin a highly precise and exact manner when precision and exactness in ARare desirable.

SUMMARY

Various embodiments are generally directed to techniques of overlaying avirtual object on a physical object in AR. A computing device mayreceive one or more images of the physical object, perform analysis onthe images to generate a digital outline of the physical object, anddetermine a position (e.g., rotation) and a scale of the physical objectbased at least in part on the generated digital outline. The computingdevice may configure (e.g., rotate, scale) a 3D model of the physicalobject to match the determined position and scale of the physicalobject. The computing device may then place or overlay a 3D virtualobject on the physical object in AR based on a predefined locationalrelation between the 3D virtual object and the 3D model of the physicalobject, and further, generate a composite view of the 3D virtual objectplaced or overlaid on the physical object.

The analysis performed to determine the position and scale of thephysical object may include performing image segmentation to generate adigital outline of the physical object. In embodiments, aguess-and-check model and/or a machine learning model may be applied tothe digital outline to determine the position and scale of the physicalobject.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates an example physical object in accordance with one ormore embodiments.

FIG. 2 illustrates an example user interface in accordance with one ormore embodiments.

FIG. 3 illustrates an example digital outline of a physical object inaccordance with one or more embodiments.

FIG. 4 illustrates configuring an example three-dimensional model of aphysical object in accordance with one or more embodiments.

FIG. 5 illustrates an example composite view in accordance with one ormore embodiments.

FIG. 6 illustrates an example flow diagram in accordance with one ormore embodiments.

FIG. 7 illustrates an example computing architecture of a computingdevice in accordance with one or more embodiments.

FIG. 8 illustrates an example communications architecture in accordancewith one or more embodiments.

DETAILED DESCRIPTION

Various embodiments are generally directed to overlaying orsuperimposing a 3D virtual object onto a physical object in AR in ahighly precise manner. In at least that regard, a 3D virtual object canbe “snapped-on” or “snapped-into” a physical object in the correctlocation, which may be particularly desirable for using AR technology onphysical objects that are modular in nature, such as a vehicle, a house,etc.

By way of example, the physical object may be a vehicle. A user may beable to preview, in AR, what a vehicle trim (e.g., spoiler) would looklike on the vehicle prior to the user installing the trim. For instance,using a computing device (e.g., smartphone, tablet computer, laptop,etc.), a user may be able to view a composite AR rendering of a virtualversion of the trim that is “snapped-on” the vehicle in the real-worldenvironment in real time and at its correct location. The user may beable to move the computing device around the vehicle and view thecomposite AR rendering while the trim continually remains in its correctposition irrespective of whether the trim is partially or completelyhidden from the current viewpoint.

According to one example, the user may indicate the specific make,model, and/or generation of the vehicle. In a different example,computer vision may be used to predict the make, model, and/orgeneration of the vehicle. Moreover, the user may specify a trim thatthe user desires to preview on the vehicle. A 3D model of the vehicle, avirtual version of the trim, and the known correct location of thevirtual trim in relation to the 3D model may be provided, generated,accessed, or determined. In examples, the 3D model of the vehicle andthe virtual trim may be created ahead of time. Further, the correctlocation of the virtual trim in relation to the 3D model may bepredefined or predetermined.

In embodiments, one or more images of the vehicle in its real-worldenvironment may be received by the computing device in real time.Analysis, such as image segmentation, may be performed on the receivedimages to locate the physical object in the image and generate a digitaloutline of the vehicle, which may be a two-dimensional (2D) outline ofthe vehicle. Based on this digital outline, the position (e.g.,rotation, spherical coordinates) and the scale of the vehicle in theenvironment may be determined.

Using the determined position and scale, the 3D model of the vehicle maybe configured (e.g., positioned, scaled) correctly. For example, a 2Doutline of the 3D model of the vehicle outline may be extracted at aparticular rotation and scale, and that 2D outline of the 3D model maybe compared to the 2D outline of the vehicle (e.g., the digital outlineof the vehicle) to determine whether the two outlines match or align.This configuration process may be repeated until the two outlines areconfigured to align and may be performed based on or using a trainedmachine learning model or a directed guess-and-check method. The virtualtrim may then be placed, superimposed, or overlaid onto the vehicle inAR based on the predefined locational relation of the virtual trim withrespect to the 3D model of the vehicle. Accordingly, the user may beable to preview what the trim looks like on the vehicle with a highdegree of reliability and accuracy before the trim is installed.

It may be understood that the term trim is applied broadly and mayinclude any add-on, option, or any suitable modification to the vehicle,as will be further described below. Moreover, while the above-describedphysical object is a vehicle, it may be understood that the physicalobject may be any real-world object, particularly objects that aremodular, such as a house so that various additions or modifications(e.g., sunroom, new garage, deck) thereon can be previewed in AR.

Previously or currently available AR technology is limited in that theprecision and exactness of overlaying virtual add-ons on physicalobjects is neither an important nor a primary objective. There are,however, various applications of AR technology where precision andexactness are highly desirable. The above-described features and therelated embodiments and examples described herein are advantageous overthe previously or currently available AR technology. For example,analysis, such as image segmentation, may be performed on images of aphysical object to generate an accurate digital outline of the physicalobject, which then may be used to accurately determine the position(e.g., rotation) and the scale of the physical object by comparing a 2Doutline of a 3D model of physical object at particular rotations(s) andscale(s) with the digital outline of the physical object based on amachine learning model and/or a guess-and-check method. By configuring a3D model of the physical object to match the determined position andscale, a virtual object may be overlaid on or “snapped on” to a physicalobject in AR based on a known locational relation between the virtualobject and the 3D model, thereby allowing high precision in the ARprocess.

Reference is now made to the drawings, where like reference numerals areused to refer to like elements throughout. In the following description,for the purpose of explanation, numerous specific details are set forthin order to provide a thorough understanding thereof. It may be evident,however, that the novel embodiments can be practiced without thesespecific details. In other instances, well-known structures and devicesare shown in block diagram form to facilitate a description thereof. Theintention is to cover all modification, equivalents, and alternativeswithin the scope of the claims.

FIG. 1 illustrates an example physical object according to one or moreembodiments. As shown, the physical object is a vehicle 100 (e.g., atwo-door sports coupe) that may be parked in an inventory lot at a cardealership 102. The vehicle 100 may have a trim level that is mostbasic, and thus, a user may be able to customize the vehicle 100 withindividual trims, various add-ons, modifications, upgrades, trimpackages, and the like. By way of example, the trims may include aspoiler, a fog lamp, a headlight, a tail light, a wheel rim, an exhaustpipe, a bumper add-on, paint color, a decal or sticker, a tint shade,etc. Prior to purchasing or installing any trim, the user may desire topreview what the trim would look like on the vehicle. Moreover, if thereare different designs or models of the same type of trim, the user maywant to see how those different designs and models look like or fit onthe vehicle. In a further example, the physical object may be a scalemodel of a vehicle, such as vehicle 100, that may be displayed in akiosk, model showroom, or any suitable display structure. Thus, a usermay be able to interact with the vehicle represented by the scale modeland preview various add-ons, modifications, upgrades, trims, etc. viathe AR-based overlay technique described herein and any other applicablecomputing technology, such as artificial intelligence (AI) technology.

In embodiments, the user may use a computing device 104, such as asmartphone, a tablet computer, a laptop, or any suitable mobilecomputing device, to place, superimpose, or overlay a virtual version ofthe trim on the vehicle in AR. As shown, the user may hold up thecomputing device 104 so that the vehicle 100 is sufficiently within afield-of-view 106 of at least one sensor of the computing device 104.For instance, the sensor may be an image sensor, a camera, an infraredcamera, a laser sensor, or any other suitable sensor for capturingimages or data related to the physical attributes of the vehicle 100.

FIG. 2 illustrates an example user interface 200 according to one ormore embodiments. The user interface 200 may be displayed on a computingdevice, which may be similar to the above-described computing device104. In embodiments, the user may be prompted by the interface 200 toenter the make and model of the vehicle. Based on this user input, thecomputing device may acquire from a storage device a 3D model of thevehicle, as will be further described below. In the illustrated example,the make of the vehicle may be a Jaguar and the model may be the F-TypeCoupe. In alternative embodiments, the computing device may receive oneor more images of the vehicle via at least one camera, and computervision processing may be performed to predict the make and model of thevehicle. Moreover, the one or more images of the vehicle may be input toan image classifier to determine the make and model. Upon determinationof the make and model, the interface 200 may also prompt the user forconfirmation.

The interface 200 may display various trims that may be superimposed onthe vehicle in AR. As illustrated, the trims include two differentmodels of a spoiler (spoiler M1 and spoiler M2), three different designsof a wheel rim (wheel rim D1, wheel rim D2, and wheel rim D3), a taillight, a paint color, a bumper add-on, a rear bumper add-on, a tintshade, an exhaust pipe, a headlight, a fog lamp, a decal or sticker. Inembodiments, the user may be able to press any lettering portion of eachtrim (or hover a cursor over the lettering portion) to view an image, ora virtual version, of the trim. The user may select one or more of thetrims by pressing the respective square icon on the left. As shown inFIG. 2, the user may select three different trims to place, superimpose,or overlay on the vehicle: the spoiler M1, the wheel rim D2, and thebumper add-on.

FIG. 3 illustrates an example digital outline 300 of a physical objectaccording to one or more embodiments. As shown, the physical object maybe a vehicle 302, which may be similar to the vehicle 100 of FIG. 1. Inexamples, a computing device may receive one or more images, e.g., image304, of the vehicle 302 via at least one camera. Alternatively, one ormore frames from a video feed of the vehicle may be received. Using theone or more images, or the one or more frames, a digital (e.g.,pixel-level) outline of the vehicle may be generated by performinganalysis on the images or the frames.

By way of example, the performed analysis may be image segmentation,which is a computer-vision-based process of partitioning a digital imageinto multiple segments (e.g., pixels, super-pixels) and used to locateobjects and boundaries (e.g., lines, curves, etc.) in the image, wherebya label to every pixel may be assigned such that pixels with the samelabel may share certain characteristics. There are numerous imagesegmentation methods: thresholding methods, clustering methods, motionand interactive segmentation, compression-based methods, histogram-basedmethods, dual clustering method, region-growing methods, partialdifferential equation-based methods, variational methods, graphpartitioning methods, watershed transformation, model-basedsegmentation, multi-scale segmentation, semi-automatic segmentation,trainable segmentation, segmentation of related images and videos, etc.One or more of these methods may be used herein. Moreover, performingimage segmentation on the images or frames may include applying aconvolutional neural network (CNN) learning algorithm (e.g., DeepLabV3,U-net) on the one or more images or the one or more frames, where apredetermined number of samples may be used to train the CNN learningalgorithm.

In further embodiments, the digital outline 300 may be a pixelated orsuper-pixelated boundary of the vehicle 302 and may be generated usingan edge detection method via an edge detector. For example, edgedetection may include various mathematical methods that identify pointsin a digital image at which the image brightness changes sharply or hasdiscontinuities. It may be understood that the digital outline 300 is a2D outline of the vehicle 302.

FIG. 4 illustrates configuring an example 3D model 400 of a physicalobject according to one or more embodiments. The physical object, again,may be a vehicle and the 3D model 400 of the vehicle may be provided,generated, accessed, or determined. As shown, the 3D model 400 may be anexact virtual replica of the vehicle. It may include all the basiccomponents, details, and trims found in the original vehicle. Inexamples, the 3D model 400 may be created ahead or time, and thus,provided to a computing device or accessed by the computing device. Inalternative examples, the 3D model 400 may be determined or generatedon-the-fly and in real-time using various types of information and datacorresponding to the vehicle, e.g., images of the vehicle, publiclyavailable data or information on the Internet, etc.

A digital outline 402 of the vehicle, which may be similar to theabove-described digital outline 300, may be used to determine theposition and the scale of the vehicle in the image or frame. Inembodiments, the determination of the position and scale of the vehiclemay include using or applying an object-specific machine learning model,which may include tree-based methods (e.g., random forest, gradientboosted machine (GBM), Classification And Regression Trees (CART)),linear or logistic regression, neural network, support vector machines(SVM), etc. that may be trained on an outline of the 3D model 400 atnumerous rotations and/or scales to the generated digital outline 402 ofthe vehicle to accurately predict the position and scale. It may beunderstood that the machine learning model may be a general “one sizefits all” model that predicts rotation and/or scale based on a 3D objectand a 2D outline and is not required to be object-specific. In anotherembodiment, the determination of the position and scale of the vehiclemay include applying a guess-and-check model that compares the outlineof the 3D model 400 at numerous rotations and/or scales for apredetermined number of guesses and checks. In at least that regard, inone or both of the above-described embodiments, a 2D outline of the 3Dmodel 400 in a particular rotation and scale may be extracted, and that2D outline of the 3D model may be compared to the 2D outline of thevehicle (e.g., the digital outline 402) to determine, or until it isdetermined, that the two outlines match or align. It may be understoodthat the term position may be broad and include information on therotation (e.g., yaw, pitch) of the vehicle as well as sphericalcoordinates (e.g., radial distance, polar angle, azimuthal angle)associated with the vehicle.

Once the specific position and scale of the vehicle are determined, the3D model 400 may be configured to match the position and scale of thevehicle by correctly positioning, rotating, and scaling the 3D model 400of the vehicle. To at least that end, the result 404 of theconfiguration process may be that the 3D model 400 of the vehicleprecisely aligns with the digital outline 402, as shown. It may beunderstood that the outlines of the 3D model 400 and the digital outline402 may not line up pixel-perfect. Thus, the outlines may be consideredto match or aligned when a desired or predetermined threshold match oralignment is achieved (e.g., to an acceptable or predetermined thresholdof error). The determination of whether the outlines match or align maybe applied similarly to the guess-and-check method in that the nextguess may be selected based on the error (e.g., outline mismatch) of oneor more previous guesses.

Upon at least correctly positioning and scaling the 3D model 400, avirtual trim (or trims) may be placed, superimposed, or overlaid on thevehicle in the correct location(s). The correct location of the virtualtrim in relation to the 3D model 400 (which may otherwise be known asthe locational relationship between the virtual trim and the 3D model400) may be predefined or predetermined. In other words, for instance,it may already be known exactly where on the vehicle the trim ispositioned, and based on this known association, the virtual trim may beeasily and correctly placed on the 3D model 400. Thus, for example, andas shown, the specific trims selected by the user in FIG. 2—the spoilerM1, the bumper add-on, and the wheel rim D2—may be correctly andprecisely placed, superimposed, or overlaid on the vehicle at positions406, 408, and 410, respectively.

Moreover, a plane detection with respect to a floor may be performed todetermine the correct location of the virtual trim on the vehicle. Byway of example, various feature points on the floor may be used tocalculate the location of the virtual trim, which may be based onknowing the position and other location information of the 3D model 400relative to a virtual floor that corresponds to the physical floor.

It may be understood in scenarios where physical objects arerotationally symmetric (e.g., vertical symmetry of a vehicle—the frontand back of the vehicle would look the same from a dead-centerviewpoint), mismatches between the 3D model of the physical object andthe digital outline may occur. Thus, in these scenarios, multiple imagesor frames may be analyzed to ensure there is no mismatch and/or the usermay be prompted to not point the computing device or look at thephysical object directly from the line of symmetry. Further, it may beunderstood that while only a single image or frame may be used todetermine the position and scale of the vehicle, and overlay a virtualtrim thereon, it may be more robust to analyze multiple images orframes, especially as the user moves the computing device around thevehicle, which would produce numerous different digital outlines,positions, rotations, and/or scales of the vehicle. Once the virtualtrim is correctly placed on the vehicle, one or more images or framesmay be periodically checked to verify that the position and scale of thereal-world vehicle still align with the 3D model of the vehicle.

FIG. 5 illustrates an example composite view 500 according to one ormore embodiments. As shown, a computing device 504 displays thecomposite view 500 of the virtual trims (e.g., the trims selected by theuser in FIG. 2—the spoiler M1, the bumper add-on, and the wheel rim D2)that are placed, superimposed, or overlaid on a vehicle 506 in AR togenerate an AR-modified vehicle 502. In this regard, the user canpreview what the selected trims will look like on the vehicle 506 inreal time via the AR-modified vehicle 502 prior to purchasing orinstalling the trims. The composite view 500 may be generated from acamera-end of a field-of-view 508.

In embodiments, the interface of the computing device 504 may allow theuser to add or remove trims or other options while simultaneouslyviewing the AR-modified vehicle 502 in the composite view 500. Forinstance, the user may decide to change the paint color of the vehiclein real time. The computing device 504 may then automatically update thepaint color such that the AR-modified vehicle 502 shows the new paintcolor selected by the user in the composite view 500.

In examples, the user may walk around the vehicle with the computingdevice 504 to view the AR-modified vehicle 502 at different angles orpositions. Thus, for instance, at least the position, rotation, scale,etc. of the 3D model of the vehicle may be continuously updated so thatthe virtual trims remain in the correct locations on the AR-modifiedvehicle 502 as the computing device is moved around. Moreover, thevirtual trims may be placed, superimposed, or overlaid on the vehicle506 irrespective of whether any of the virtual trims are partially orcompletely visible in the relevant field-of-view. As shown in FIG. 5,for instance, the spoiler is partially visible on the AR-modifiedvehicle 502 when in the field-of-view 508. This feature is advantageousand desirable because it allows the user to see what the spoiler lookslike from that viewpoint despite the spoiler being partially hidden.

It may be understood that the above-described features and techniquesfor overlaying virtual objects in a precise and exact manner may beapplied to any real-world object, particularly objects that are modularin nature, for example, previewing renovation-related add-ons for ahouse. Other examples may also include interior design, construction,engineering applications, gaming, etc.

FIG. 6 illustrates a flow diagram 600 in accordance with one or moreembodiments. It may be understood that the features associated with theillustrated blocks may be performed or executed by one or more computingdevices and/or processing circuitry contained therein.

At block 602, one or more images of a physical object may be received.The physical object may be any real-world object, such as a vehicle, ahouse, a boat, an airplane, etc. As described above, the one or moreimages may be captured by a camera or any suitable imaging devicecoupled to a computing device. For instance, a user may hold up thecamera and take pictures or a video of the physical object.

At block 604, an analysis is performed on the one or more images togenerate a digital outline of the physical object. As set forth above,the analysis may include performing image segmentation. Additionally, oralternatively, the analysis may include performing an edge detection viaan edge detector to generate a pixelated or super-pixelated outline ofthe object. The digital outline may be a 2D outline of the physicalobject.

At block 606, based on the generated digital outline of the physicalobject, a position and a scale of the object may be determined orpredicted, which may be based on different techniques. For example, a 3Dmodel of the physical object, which may have been created beforehand,may be provided, accessed, or generated. A machine learning model thatis trained on an outline of the 3D model of the object at one or moredifferent rotations and/or scales may be applied to the digital outlinegenerated at block 604. The machine learning model may then accuratelypredict the position and scale of the physical object. The machinelearning model may include tree-based methods (e.g., random forest,gradient boosted machine (GBM), Classification And Regression Trees(CART)), linear or logistic regression, neural network, support vectormachines (SVM), etc. In another example, a guess-and-check model may beemployed, which compares the outline of the 3D model of the physicalobject at different rotations and scales for a predetermined number ofguesses and/or checks, and subsequently predicts or determines theposition and scale of the physical object once the guess-and-check modelachieves a desired result. In either or both examples, as set forthabove, the 2D outline of the 3D model of the physical object in aparticular rotation and scale may be extracted, and that 2D outline ofthe 3D model may be compared to the 2D outline of the physical object(e.g., the digital outline generated at block 606).

At block 608, the 3D model of the physical object may be configured,e.g., positioned, rotated, scaled, etc., to match the determinedposition and scale at block 606. The configuration of the 3D modelallows a 3D virtual object, such as a vehicle trim or an addition to ahouse, to be correctly placed, superimposed, or overlaid on the physicalobject.

At block 610, the virtual object is placed or overlaid, or “snapped,” onthe physical object in AR based on a known locational relation betweenthe 3D virtual object and the 3D model of the physical object. Inembodiments, the 3D model of the physical object and all relevant orpossible virtual add-ons may be created beforehand, which provides theblueprint for correctly placing a virtual object on the physical objectin AR in real time.

At block 612, a composite view may be digitally rendered or generated,which includes the 3D virtual object correctly and accurately placed oroverlaid on the physical object. It may be understood that the processdescribed in blocks 602 to 612 may be repeated for every image or videoframe the computing device receives, thereby providing a continuous,updated real-time preview of the 3D virtual object in AR even as thecomputing device is moving around the physical object.

It may be understood that the blocks illustrated in FIG. 6 are notlimited to any specific order. One or more of the blocks may beperformed or executed simultaneously or near simultaneously. Forexample, the hashing and salting of the key may be performed at the sametime.

FIG. 7 illustrates an embodiment of an exemplary computing architecture700, e.g., of a computing device, such as a desktop computer, laptop,tablet computer, mobile computer, smartphone, etc., suitable forimplementing various embodiments as previously described. The computingdevice may be the computing devices 104, 200, and 504 illustrated inFIGS. 1, 2 and 5, respectively. In one embodiment, the computingarchitecture 700 may include or be implemented as part of a system,which will be further described below. As described above, at least onecomputing device and/or the processing circuitries thereof may beconfigured to at least receive one or more images (or frames of a videofeed) of a physical object via at least one camera (which may be coupledto or integrated in the computing device) to perform analysis (e.g.,image segmentation) on the images, and further determine the positionand scale of the physical object to overlay a virtual object on thephysical object at the correct location.

As used in this application, the terms “system” and “component” areintended to refer to a computer-related entity, either hardware, acombination of hardware and software, software, or software inexecution, examples of which are provided by the exemplary computingarchitecture 700. For example, a component can be, but is not limited tobeing, a process running on a processor, a processor, a hard disk drive,multiple storage drives (of optical and/or magnetic storage medium), anobject, an executable, a thread of execution, a program, and/or acomputer. By way of illustration, both an application running on aserver and the server can be a component. One or more components canreside within a process and/or thread of execution, and a component canbe localized on one computer and/or distributed between two or morecomputers. Further, components may be communicatively coupled to eachother by various types of communications media to coordinate operations.The coordination may involve the uni-directional or bi-directionalexchange of information. For instance, the components may communicateinformation in the form of signals communicated over the communicationsmedia. The information can be implemented as signals allocated tovarious signal lines. In such allocations, each message is a signal.Further embodiments, however, may alternatively employ data messages.Such data messages may be sent across various connections. Exemplaryconnections include parallel interfaces, serial interfaces, and businterfaces.

The computing architecture 700 includes various common computingelements, such as one or more processors, multi-core processors,co-processors, memory units, chipsets, controllers, peripherals,interfaces, oscillators, timing devices, video cards, audio cards,multimedia input/output (I/O) components, power supplies, and so forth.The embodiments, however, are not limited to implementation by thecomputing architecture 700.

As shown in FIG. 7, the computing architecture 700 includes processor704, a system memory 706 and a system bus 708. The processor 704 can beany of various commercially available processors, processing circuitry,central processing unit (CPU), a dedicated processor, afield-programmable gate array (FPGA), etc.

The system bus 708 provides an interface for system componentsincluding, but not limited to, the system memory 706 to the processor704. The system bus 708 can be any of several types of bus structurethat may further interconnect to a memory bus (with or without a memorycontroller), a peripheral bus, and a local bus using any of a variety ofcommercially available bus architectures. Interface adapters may connectto the system bus 708 via slot architecture. Example slot architecturesmay include without limitation Accelerated Graphics Port (AGP), CardBus, (Extended) Industry Standard Architecture ((E)ISA), Micro ChannelArchitecture (MCA), NuBus, Peripheral Component Interconnect (Extended)(PCI(X)), PCI Express, Personal Computer Memory Card InternationalAssociation (PCMCIA), and the like.

The computing architecture 700 may include or implement various articlesof manufacture. An article of manufacture may include acomputer-readable storage medium to store logic. Examples of acomputer-readable storage medium may include any tangible media capableof storing electronic data, including volatile memory or non-volatilememory, removable or non-removable memory, erasable or non-erasablememory, writeable or re-writeable memory, and so forth. Examples oflogic may include executable computer program instructions implementedusing any suitable type of code, such as source code, compiled code,interpreted code, executable code, static code, dynamic code,object-oriented code, visual code, and the like. Embodiments may also beat least partly implemented as instructions contained in or on anon-transitory computer-readable medium, which may be read and executedby one or more processors to enable performance of the operationsdescribed herein.

The system memory 706 may include various types of computer-readablestorage media in the form of one or more higher speed memory units, suchas read-only memory (ROM), random-access memory (RAM), dynamic RAM(DRAM), Double-Data-Rate DRAM (DDRAM), synchronous DRAM (SDRAM), staticRAM (SRAM), programmable ROM (PROM), erasable programmable ROM (EPROM),electrically erasable programmable ROM (EEPROM), flash memory, polymermemory such as ferroelectric polymer memory, ovonic memory, phase changeor ferroelectric memory, silicon-oxide-nitride-oxide-silicon (SONOS)memory, magnetic or optical cards, an array of devices such as RedundantArray of Independent Disks (RAID) drives, solid state memory devices(e.g., USB memory, solid state drives (SSD) and any other type ofstorage media suitable for storing information. In the illustratedembodiment shown in FIG. 7, the system memory 706 can includenon-volatile memory 710 and/or volatile memory 712. A basic input/outputsystem (BIOS) can be stored in the non-volatile memory 710.

The computer 702 may include various types of computer-readable storagemedia in the form of one or more lower speed memory units, including aninternal (or external) hard disk drive (HDD) 714, a magnetic floppy diskdrive (FDD) 716 to read from or write to a removable magnetic disk 718,and an optical disk drive 720 to read from or write to a removableoptical disk 722 (e.g., a CD-ROM or DVD). The HDD 714, FDD 716 andoptical disk drive 720 can be connected to the system bus 708 by a HDDinterface 724, an FDD interface 726 and an optical drive interface 728,respectively. The HDD interface 724 for external drive implementationscan include at least one or both of Universal Serial Bus (USB) and IEEE1394 interface technologies.

The drives and associated computer-readable media provide volatileand/or nonvolatile storage of data, data structures, computer-executableinstructions, and so forth. For example, a number of program modules canbe stored in the drives and memory units 710, 712, including anoperating system 730, one or more application programs 732, otherprogram modules 734, and program data 736. In one embodiment, the one ormore application programs 732, other program modules 734, and programdata 736 can include, for example, the various applications and/orcomponents of the system 800.

A user can enter commands and information into the computer 702 throughone or more wire/wireless input devices, for example, a keyboard 738 anda pointing device, such as a mouse 740. Other input devices may includemicrophones, infra-red (IR) remote controls, radio-frequency (RF) remotecontrols, game pads, stylus pens, card readers, dongles, finger printreaders, gloves, graphics tablets, joysticks, keyboards, retina readers,touch screens (e.g., capacitive, resistive, etc.), trackballs, trackpads, sensors, styluses, and the like. These and other input devices areoften connected to the processor 704 through an input device interface742 that is coupled to the system bus 708 but can be connected by otherinterfaces such as a parallel port, IEEE 1394 serial port, a game port,a USB port, an IR interface, and so forth.

A monitor 744 or other type of display device is also connected to thesystem bus 708 via an interface, such as a video adaptor 746. Themonitor 744 may be internal or external to the computer 702. In additionto the monitor 744, a computer typically includes other peripheraloutput devices, such as speakers, printers, and so forth.

The computer 702 may operate in a networked environment using logicalconnections via wire and/or wireless communications to one or moreremote computers, such as a remote computer 748. The remote computer 748can be a workstation, a server computer, a router, a personal computer,portable computer, microprocessor-based entertainment appliance, a peerdevice or other common network node, and typically includes many or allthe elements described relative to the computer 702, although, forpurposes of brevity, only a memory/storage device 750 is illustrated.The logical connections depicted include wire/wireless connectivity to alocal area network (LAN) 752 and/or larger networks, for example, a widearea network (WAN) 754. Such LAN and WAN networking environments arecommonplace in offices and companies, and facilitate enterprise-widecomputer networks, such as intranets, all of which may connect to aglobal communications network, for example, the Internet.

When used in a LAN networking environment, the computer 702 is connectedto the LAN 752 through a wire and/or wireless communication networkinterface or adaptor 756. The adaptor 756 can facilitate wire and/orwireless communications to the LAN 752, which may also include awireless access point disposed thereon for communicating with thewireless functionality of the adaptor 756.

When used in a WAN networking environment, the computer 702 can includea modem 758, or is connected to a communications server on the WAN 754or has other means for establishing communications over the WAN 754,such as by way of the Internet. The modem 758, which can be internal orexternal and a wire and/or wireless device, connects to the system bus708 via the input device interface 742. In a networked environment,program modules depicted relative to the computer 702, or portionsthereof, can be stored in the remote memory/storage device 750. It willbe appreciated that the network connections shown are exemplary andother means of establishing a communications link between the computerscan be used.

The computer 702 is operable to communicate with wire and wirelessdevices or entities using the IEEE 802 family of standards, such aswireless devices operatively disposed in wireless communication (e.g.,IEEE 802.11 over-the-air modulation techniques). This includes at leastWi-Fi (or Wireless Fidelity), WiMax, and Bluetooth™ wirelesstechnologies, among others. Thus, the communication can be a predefinedstructure as with a conventional network or simply an ad hoccommunication between at least two devices. Wi-Fi networks use radiotechnologies called IEEE 802.118 (a, b, g, n, etc.) to provide secure,reliable, fast wireless connectivity. A Wi-Fi network can be used toconnect computers to each other, to the Internet, and to wire networks(which use IEEE 802.3-related media and functions).

The various elements of the devices as previously described withreference to FIGS. 1-6 may include various hardware elements, softwareelements, or a combination of both. Examples of hardware elements mayinclude devices, logic devices, components, processors, microprocessors,circuits, processors, circuit elements (e.g., transistors, resistors,capacitors, inductors, and so forth), integrated circuits, applicationspecific integrated circuits (ASIC), programmable logic devices (PLD),digital signal processors (DSP), field programmable gate array (FPGA),memory units, logic gates, registers, semiconductor device, chips,microchips, chip sets, and so forth. Examples of software elements mayinclude software components, programs, applications, computer programs,application programs, system programs, software development programs,machine programs, operating system software, middleware, firmware,software modules, routines, subroutines, functions, methods, procedures,software interfaces, application program interfaces (API), instructionsets, computing code, computer code, code segments, computer codesegments, words, values, symbols, or any combination thereof. However,determining whether an embodiment is implemented using hardware elementsand/or software elements may vary in accordance with any number offactors, such as desired computational rate, power levels, heattolerances, processing cycle budget, input data rates, output datarates, memory resources, data bus speeds and other design or performanceconstraints, as desired for a given implementation.

FIG. 8 is a block diagram depicting an exemplary communicationsarchitecture 800 suitable for implementing various embodiments. Forexample, one or more computing devices may communicate with each othervia a communications framework, such as a network. At least onecomputing devices connected to the network may be a user computingdevice, such as a desktop computer, laptop, tablet computer, smartphone,etc. (e.g., computing devices 104, 200, and 504 illustrated in FIGS. 1,2 and 5, respectively). At least a second computing device connected tothe network may be one or more back-end server computers. In someembodiments, the user computing device may be configured to send theserver computers one or more images for performing the overlay ofvirtual objects in AR and providing a digital composite view back to theuser computing device.

The communications architecture 800 includes various commoncommunications elements, such as a transmitter, receiver, transceiver,radio, network interface, baseband processor, antenna, amplifiers,filters, power supplies, and so forth. The embodiments, however, are notlimited to implementation by the communications architecture 800.

As shown in FIG. 8, the communications architecture 800 includes one ormore clients 802 and servers 804. The one or more clients 802 and theservers 804 are operatively connected to one or more respective clientdata stores 806 and server data stores 807 that can be employed to storeinformation local to the respective clients 802 and servers 804, such ascookies and/or associated contextual information. By way of example,server data store 807 may store all hashed and salted biometric keys.

The clients 802 and the servers 804 may communicate information betweeneach other using a communication framework 810. The communicationsframework 810 may implement any well-known communications techniques andprotocols. The communications framework 810 may be implemented as apacket-switched network (e.g., public networks such as the Internet,private networks such as an enterprise intranet, and so forth), acircuit-switched network (e.g., the public switched telephone network),or a combination of a packet-switched network and a circuit-switchednetwork (with suitable gateways and translators).

The communications framework 810 may implement various networkinterfaces arranged to accept, communicate, and connect to acommunications network. A network interface may be regarded as aspecialized form of an input/output (I/O) interface. Network interfacesmay employ connection protocols including without limitation directconnect, Ethernet (e.g., thick, thin, twisted pair 10/100/1000 Base T,and the like), token ring, wireless network interfaces, cellular networkinterfaces, IEEE 802.7a-x network interfaces, IEEE 802.16 networkinterfaces, IEEE 802.20 network interfaces, and the like. Further,multiple network interfaces may be used to engage with variouscommunications network types. For example, multiple network interfacesmay be employed to allow for the communication over broadcast,multicast, and unicast networks. Should processing requirements dictatea greater amount speed and capacity, distributed network controllerarchitectures may similarly be employed to pool, load balance, andotherwise increase the communicative bandwidth required by clients 802and the servers 804. A communications network may be any one and thecombination of wired and/or wireless networks including withoutlimitation a direct interconnection, a secured custom connection, aprivate network (e.g., an enterprise intranet), a public network (e.g.,the Internet), a Personal Area Network (PAN), a Local Area Network(LAN), a Metropolitan Area Network (MAN), an Operating Missions as Nodeson the Internet (OMNI), a Wide Area Network (WAN), a wireless network, acellular network, and other communications networks.

The components and features of the devices described above may beimplemented using any combination of discrete circuitry, applicationspecific integrated circuits (ASICs), logic gates and/or single chiparchitectures. Further, the features of the devices may be implementedusing microcontrollers, programmable logic arrays and/or microprocessorsor any combination of the foregoing where suitably appropriate. It isnoted that hardware, firmware and/or software elements may becollectively or individually referred to herein as “logic” or “circuit.”

At least one computer-readable storage medium may include instructionsthat, when executed, cause a system to perform any of thecomputer-implemented methods described herein.

Some embodiments may be described using the expression “one embodiment”or “an embodiment” along with their derivatives. These terms mean that aparticular feature, structure, or characteristic described in connectionwith the embodiment is included in at least one embodiment. Theappearances of the phrase “in one embodiment” in various places in thespecification are not necessarily all referring to the same embodiment.Moreover, unless otherwise noted the features described above arerecognized to be usable together in any combination. Thus, any featuresdiscussed separately may be employed in combination with each otherunless it is noted that the features are incompatible with each other.

With general reference to notations and nomenclature used herein, thedetailed descriptions herein may be presented in terms of programprocedures executed on a computer or network of computers. Theseprocedural descriptions and representations are used by those skilled inthe art to most effectively convey the substance of their work to othersskilled in the art.

A procedure is here, and generally, conceived to be a self-consistentsequence of operations leading to a desired result. These operations arethose requiring physical manipulations of physical quantities. Usually,though not necessarily, these quantities take the form of electrical,magnetic or optical signals capable of being stored, transferred,combined, compared, and otherwise manipulated. It proves convenient attimes, principally for reasons of common usage, to refer to thesesignals as bits, values, elements, symbols, characters, terms, numbers,or the like. It should be noted, however, that all of these and similarterms are to be associated with the appropriate physical quantities andare merely convenient labels applied to those quantities.

Further, the manipulations performed are often referred to in terms,such as adding or comparing, which are commonly associated with mentaloperations performed by a human operator. No such capability of a humanoperator is necessary, or desirable in most cases, in any of theoperations described herein, which form part of one or more embodiments.Rather, the operations are machine operations.

Some embodiments may be described using the expression “coupled” and“connected” along with their derivatives. These terms are notnecessarily intended as synonyms for each other. For example, someembodiments may be described using the terms “connected” and/or“coupled” to indicate that two or more elements are in direct physicalor electrical contact with each other. The term “coupled,” however, mayalso mean that two or more elements are not in direct contact with eachother, but yet still co-operate or interact with each other.

Various embodiments also relate to apparatus or systems for performingthese operations. This apparatus may be specially constructed for therequired purpose and may be selectively activated or reconfigured by acomputer program stored in the computer. The procedures presented hereinare not inherently related to a particular computer or other apparatus.The required structure for a variety of these machines will appear fromthe description given.

It is emphasized that the Abstract of the Disclosure is provided toallow a reader to quickly ascertain the nature of the technicaldisclosure. It is submitted with the understanding that it will not beused to interpret or limit the scope or meaning of the claims. Inaddition, in the foregoing Detailed Description, it can be seen thatvarious features are grouped together in a single embodiment for thepurpose of streamlining the disclosure. This method of disclosure is notto be interpreted as reflecting an intention that the claimedembodiments require more features than are expressly recited in eachclaim. Rather, as the following claims reflect, inventive subject matterlies in less than all features of a single disclosed embodiment. Thus,the following claims are hereby incorporated into the DetailedDescription, with each claim standing on its own as a separateembodiment. In the appended claims, the terms “including” and “in which”are used as the plain-English equivalents of the respective terms“comprising” and “wherein,” respectively. Moreover, the terms “first,”“second,” “third,” and so forth, are used merely as labels, and are notintended to impose numerical requirements on their objects.

What has been described above includes examples of the disclosedarchitecture. It is, of course, not possible to describe everyconceivable combination of components and/or methodologies, but one ofordinary skill in the art may recognize that many further combinationsand permutations are possible. Accordingly, the novel architecture isintended to embrace all such alterations, modifications and variationsthat fall within the spirit and scope of the appended claims.

What is claimed is:
 1. A computer-implemented method, comprising:determining, by a processor, a position and a scale of a physical objectdepicted in an image from a viewpoint; identifying, by the processor, asymmetry-based mismatch between a three-dimensional (3D) model of thephysical object and a digital outline of the physical object at theviewpoint, the symmetry-based mismatch based at least in part onsymmetrical characteristics of the physical object; causing, by theprocessor, the 3D model of the physical object to match the position andthe scale of the physical object based on the symmetry-based mismatch;overlaying, by the processor, a 3D virtual object at a first location onan exterior portion of the physical object in augmented reality based ona predetermined locational relation between the 3D virtual object andthe 3D model of the physical object; and generating, by the processor, acomposite view of the 3D virtual object overlaid on the exterior portionof the physical object, wherein the 3D virtual object is partiallyobstructed from view by the physical object at the viewpoint, whereinthe 3D virtual object is an exterior object and a portion of the 3Dvirtual object is omitted from the composite view based on the portionof the 3D virtual object being partially obstructed by the physicalobject at the viewpoint.
 2. The computer-implemented method of claim 1,wherein the predetermined locational relation defines where the 3Dvirtual object belongs on the 3D model, the method further comprising:displaying the composite view of the 3D virtual object overlaid on theexterior portion of the physical object on a display.
 3. Thecomputer-implemented method of claim 1, further comprising: applying, bythe processor, a machine learning (ML) model to the digital outline,wherein the machine learning model is trained on a two-dimensional (2D)outline of a three-dimensional (3D) model of the physical object at aplurality of rotations; extracting, by the ML model, the 2D outline ofthe 3D model at each of the plurality of rotations of the physicalobject; comparing, by the ML model, the 2D outline of the 3D model tothe digital outline of the physical object at each of the plurality ofrotations of the physical object; and determining, by the ML model, amatch between the 2D outline of the 3D model and the digital outline ata first rotation of the plurality of rotations of the physical object.4. The computer-implemented method of claim 3, wherein the position andthe scale of the physical object is determined based on the matchbetween the 2D outline of the 3D model and the digital outline at thefirst rotation of the plurality of rotations of the physical object. 5.The computer-implemented method of claim 3, wherein the 3D virtualobject is rotated according to a first rotation of a plurality ofrotations of the 3D virtual object, wherein the 3D virtual object at thefirst rotation of the plurality of rotations of the 3D virtual objectmatches the position and scale of the physical object at the firstrotation of the plurality of rotations of the physical object.
 6. Thecomputer-implemented method of claim 3, wherein the ML model comprises aguess-and-check model, wherein a count of the plurality of rotations ofthe physical object is based on a predetermined count of guesses andchecks applied by the guess-and-check model.
 7. The computer-implementedmethod of claim 1, further comprising: detecting, by the processor, aplane in the image; detecting, by the processor, a floor in the imagebased on the plane detection; and wherein the position of the physicalobject is based at least in part on one or more feature points on thefloor and a location of the 3D model of the physical object relative toa virtual floor corresponding to the floor.
 8. The computer-implementedmethod of claim 1, further comprising: partitioning, by the processor,the image into a plurality of segments; applying, by the processor, arespective label to each of a plurality of pixels in each of theplurality of segments; locating, by the processor, the physical objectbased on the labels; and determining, by the processor, the digitaloutline of the physical object based on the labels.
 9. A non-transitorycomputer-readable storage medium, the computer-readable storage mediumincluding instructions that when executed by a processor, cause theprocessor to: determine a position and a scale of a physical objectdepicted in an image from a viewpoint; identify a symmetry-basedmismatch between a three-dimensional (3D) model of the physical objectand a digital outline of the physical object at the viewpoint, thesymmetry-based mismatch based at least in part on symmetricalcharacteristics of the physical object; cause the 3D model of thephysical object to match the position and the scale of the physicalobject based on the symmetry-based mismatch; overlay a 3D virtual objectat a first location on an exterior portion of the physical object inaugmented reality based on a predetermined locational relation betweenthe 3D virtual object and the 3D model of the physical object; andgenerate a composite view of the 3D virtual object overlaid on theexterior portion of the physical object, wherein the 3D virtual objectis partially obstructed from view by the physical object at theviewpoint, wherein the 3D virtual object is an exterior object and aportion of the 3D virtual object is omitted from the composite viewbased on the portion of the 3D virtual object being partially obstructedby the physical object at the viewpoint.
 10. The computer-readablestorage medium of claim 9, wherein the predetermined locational relationdefines where the 3D virtual object belongs on the 3D model, wherein theinstructions further cause the processor to: display the composite viewof the 3D virtual object overlaid on the exterior portion of thephysical object on a display.
 11. The computer-readable storage mediumof claim 9, wherein the instructions further cause the processor to:apply a machine learning (ML) model to the digital outline, wherein themachine learning model is trained on a two-dimensional (2D) outline of athree-dimensional (3D) model of the physical object at a plurality ofrotations; extract, by the ML model, the 2D outline of the 3D model ateach of the plurality of rotations of the physical object; compare, bythe ML model, the 2D outline of the 3D model to the digital outline ofthe physical object at each of the plurality of rotations of thephysical object; and determine, by the ML model, a match between the 2Doutline of the 3D model and the digital outline at a first rotation ofthe plurality of rotations of the physical object.
 12. Thecomputer-readable storage medium of claim 11, wherein the position andthe scale of the physical object is determined based on the matchbetween the 2D outline of the 3D model and the digital outline at thefirst rotation of the plurality of rotations of the physical object,wherein the 3D virtual object is rotated according to a first rotationof a plurality of rotations of the 3D virtual object, wherein the 3Dvirtual object at the first rotation of the plurality of rotations ofthe 3D virtual object matches the position and scale of the physicalobject at the first rotation of the plurality of rotations of thephysical object.
 13. The computer-readable storage medium of claim 9,wherein the instructions further cause the processor to: detect, by theprocessor, a plane in the image; detect, by the processor, a floor inthe image based on the plane detection; and wherein the position of thephysical object is based at least in part on one or more feature pointson the floor and a location of the 3D model of the physical objectrelative to a virtual floor corresponding to the floor.
 14. Thecomputer-readable storage medium of claim 9, wherein the instructionsfurther cause the processor to: partition, by the processor, the imageinto a plurality of segments; apply, by the processor, a respectivelabel to each of a plurality of pixels in each of the plurality ofsegments; locate, by the processor, the physical object based on thelabels; and determine, by the processor, the digital outline of thephysical object based on the labels.
 15. A computing apparatuscomprising: a processor; and a memory storing instructions that, whenexecuted by the processor, cause the processor to: determine a positionand a scale of a physical object depicted in an image from a viewpoint;identify a symmetry-based mismatch between a three-dimensional (3D)model of the physical object and a digital outline of the physicalobject at the viewpoint, the symmetry-based mismatch based at least inpart on symmetrical characteristics of the physical object; cause the 3Dmodel of the physical object to match the position and the scale of thephysical object based on the symmetry-based mismatch; overlay a 3Dvirtual object at a first location on an exterior portion of thephysical object in augmented reality based on a predetermined locationalrelation between the 3D virtual object and the 3D model of the physicalobject; and generate a composite view of the 3D virtual object overlaidon the exterior portion of the physical object, wherein the 3D virtualobject is partially obstructed from view by the physical object at theviewpoint, wherein the 3D virtual object is an exterior object and aportion of the 3D virtual object is omitted from the composite viewbased on the portion of the 3D virtual object being partially obstructedby the physical object at the viewpoint.
 16. The computing apparatus ofclaim 15, wherein the predetermined locational relation defines wherethe 3D virtual object belongs on the 3D model, wherein the instructionsfurther cause the processor to: display the composite view of the 3Dvirtual object overlaid on the exterior portion of the physical objecton a display.
 17. The computing apparatus of claim 15, wherein theinstructions further cause the processor to: apply a machine learning(ML) model to the digital outline, wherein the machine learning model istrained on a two-dimensional (2D) outline of a three-dimensional (3D)model of the physical object at a plurality of rotations; extract, bythe ML model, the 2D outline of the 3D model at each of the plurality ofrotations of the physical object; compare, by the ML model, the 2Doutline of the 3D model to the digital outline of the physical object ateach of the plurality of rotations of the physical object; anddetermine, by the ML model, a match between the 2D outline of the 3Dmodel and the digital outline at a first rotation of the plurality ofrotations of the physical object.
 18. The computing apparatus of claim17, wherein the position and the scale of the physical object isdetermined based on the match between the 2D outline of the 3D model andthe digital outline at the first rotation of the plurality of rotationsof the physical object, wherein the 3D virtual object is rotatedaccording to a first rotation of a plurality of rotations of the 3Dvirtual object, wherein the 3D virtual object at the first rotation ofthe plurality of rotations of the 3D virtual object matches the positionand scale of the physical object at the first rotation of the pluralityof rotations of the physical object.
 19. The computing apparatus ofclaim 15, wherein the instructions further cause the processor to:detect, by the processor, a plane in the image; detect, by theprocessor, a floor in the image based on the plane detection; andwherein the position of the physical object is based at least in part onone or more feature points on the floor and a location of the 3D modelof the physical object relative to a virtual floor corresponding to thefloor.
 20. The computing apparatus of claim 15, wherein the instructionsfurther cause the processor to: partition, by the processor, the imageinto a plurality of segments; apply, by the processor, a respectivelabel to each of a plurality of pixels in each of the plurality ofsegments; locate, by the processor, the physical object based on thelabels; and determine, by the processor, the digital outline of thephysical object based on the labels.